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OO ■ Abstract 

In the 60's, Knuth introduced stack-sorting and serial compositions of stacks. In par- 
ticular, one significant question arise out of the work of Knuth: how to decide efficiently if 
■ a given permutation is sortable with 2 stacks in series? Whether this problem is polyno- 

mial or NP-complete is still unanswered yet. In this article we introduce 2-stack pushall 
permutations which form a subclass of 2-stack sortable permutations and show that these 
two classes are closely related. Moreover, we give an optimal 0(n 2 ) algorithm to decide 
if a given permutation of size n is 2-stack pushall sortable and describe all its sortings. 
This result is a step to the solve the general 2-stack sorting problem in polynomial time. 
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> ! 1 Introduction 

In the 60's, Knuth introduced the problem of stack-sorting [8] and then serial compositions of 
stacks [9]. To answer the one-stack case, he introduced both the pattern-containment relation 
^ ■ on permutations and permutation classes, two new fields of combinatorics. Stack-sorting was 

further generalized to sorting networks by Tarjan [12] while several variants appear by either 
considering other types of combinatorial structures or by changing rules \7\ IT]. 

In this article, we focus on sorting with two stacks in series. More precisely, if a is a 
permutation, we consider a as a sequence of integers ci, . . . , a n that we take as input and 
^ ! at each step we have three possibilities as described in Figure [2] (p|4j) : 

H . 

p: Get the next element of a and push its value on top of the first stack denoted H. 

A: Pop the topmost element of stack H and push this value on top of the second stack V. 

fi: Pop the topmost element of V and write it to the output. 

We iterate over these three possibilities until all elements have been output. If there is a 
sequence of operations that leads to identity on the output, then we say that the permutation 
is 2-stack sortable. Three natural questions among others arise: 

1. Decision: what is the complexity of the problem consisting of deciding whether a given 
permutation is sortable or not? 

2. Characterization: can one characterize permutations that are sortable? 

3. Counting: establish the generating function of sortable permutations. 
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For the one-stack case these three problems were solved by Knuth in [8j. A greedy algo- 
rithm allows to answer the decision problem in linear time. Moreover he characterized sortable 
permutations by introducing the 231-avoiding permutations class, whose generating function 
is the Catalan series. Since this article, the more general question of sorting with multiple 
stacks in series or in parallel has been widely studied. Knuth [5], Tarjan [T2] and Pratt [11] 
noted that the permutations sortable by the various configurations could be described by 
forbidding certain patterns to occur in the permutations. 

Regarding t parallel stacks, the decision problem can be answered in time O(nlogn) for 
t = 1,2,3, while for t > 3 this is NP-complete (this is proved by a reduction in [7] to a 
problem solved in [TO]). The characterization problem is studied in [llj: for t > 1, the basis 
of the class of permutations sortable with t stacks in parallel is infinite. Finally, about the 
counting problem, when t = 2 the generating function is described in [3], but by an infinite 
system of equations. 



For stacks in series, it has been shown in [8] that every permutation of size n can be 
sorted by log 2 (re) stacks in series. But none of the above three questions has been answered 
for more than one stack in series. For two stacks, Murphy [10] proved that the basis of the 
class of sortable permutations is infinite. In his Phd thesis, he also studies the problem of 
deciding whether a given permutation is sortable with 2 stacks in series. He reduced this 
problem to a 3-SAT problem; at the same time he reduced a 2-SAT instance to the decision 
problem, and hoped than one of both reduction was actually an equivalence. But none of 
those results has been proved or disproved. In [5], Bona gives an overview of advances in 
sorting networks and mentions this problem as possibly NP-complete. More surprising, both 
conjectures exist: in [3], the authors conjectured that the decidability problem is NP-complete, 
while Murphy in his PhD ([TO] Conjecture 260) conjectured that it is in P. Several weaker 
variants of this problem have been studied. First, West considered permutations sortable with 
two consecutive greedy passes through a stack in |14[ 115] . He conjectured the enumeration 
formula which was proved after by Zeilberger [TO]. For more than two passes, few results are 
known j6j[T7]. Another variant studied in [3] is to consider decreasing stacks (i.e. elements in 
the stack must be decreasing from bottom to top) instead of general stacks. In this article we 
define a new restriction of 2-stacks sorting, namely 2-stacks pushall sorting, and prove that 
the decidability problem in this case is polynomial. 

Throughout this article we usually write permutations as a = o\(32 ■ ■ ■ °n where n is the 
size of a, denoted by |<r|, and cij is the image of i for all i < n. A permutation tt = kyki . . . ir^ is 
a pattern of a if and only if there exist 1 < %\ < i<i < is < . . . < iy. < n such that a^a^ . . . <Ji k 
is order isomorphic to tt. We note Av{B) the set of permutations avoiding B, i.e. not having 
any permutation of B as a pattern. A permutation class C is a set of permutations downward- 
closed for the pattern relation: if a belongs to C, then every pattern of a belongs to C. Note 
that for any set B, Av(B) is a class. A permutation class C can be defined by its minimal 
set B such that C = Av(B). This minimal set is called the basis of the class. For example, 
Knuth proved that 1-stack sortable permutations are those that belong to the class Av(23T). 
Unfortunately, the basis can be infinite. For 2-stack sortable permutations, as stated above, 
it has been proved in [10] that the basis is infinite. 

A permutation can also be represented by its diagram, consisting in the set of points 
at coordinates (£, cr^) drawn in the plane (see two examples in Figure [T]). An interval in a 
permutation is a consecutive range of elements, consecutive both in indices and values. For 
example in the permutation 47968132 5, the elements 79 6 8 form an interval: they are 
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consecutive in the permutation and the values span the whole integer interval [6... 9]. In 
the diagram, note that an interval is a square which is itself a diagram of a permutation (if 
translated to the origin). In particular, no point outside this square has the same x or y 
coordinate than any cell of the square (see the yellow stripes of Figure [1]). A permutation 
where all intervals are trivial -either a singleton or the whole permutation- is called a simple 
permutation. For instance, 2 413 and 314 2 are the two simple permutations of size 4. An 
inflation of an element <7j in a by a permutation tt is the permutation obtained by replacing 
dj by 7r and renormalizing the resulting permutation. For example if we inflate 3 in 2 3 1 4 by 
the permutation 415 2 3, we obtain the permutation 26374518 (see the second diagram of 
Figure [I]). Notice that in an inflation by ir, elements corresponding to tt form an interval in 
the resulting permutation. 



Figure 1: Diagram of 479681325 and the inflation of 3 in 2314 by 41523 



We denote inflations by a = t^ 1 ), tt^ 2 \ . . . , ir^] where r is a permutation of size k and r% 
is inflated by 7r® for all i. When t is the identity 1 2 ... k (resp. the decreasing permutation 
k ... 1) we write a = ©[ttW , tt^ , . . . , tt^] (resp. ©[ttW , vr( 2 ), . . . , tt^}). 

A permutation a is ©-decomposable (resp. 0-decomposable) if it can be written a = 
©[vrW,^ 2 ),^ 3 ),...,^)] (resp. ©[vr^, tt^ 2 \ tt®, . . . , vr^]) with k > 1. Otherwise a is ffi- 
indecomposable (resp. 0-indecomposable) 

A decomposition theorem [2 J states that any permutation a ^ 1 can be written in a unique 
way as either: 

• a = ©[ttW, 7r( 2 ), 7r( 3 ), . . . ,7r( fc )] where k > 2 and the are ©-indecomposable. 

• a = ©[tt^ 1 ), ir( 2 \ 7r( 3 ), . . . ,7r^] where k > 2 and the are ©-indecomposable. 

• a = t[tt^\ 7r( 2 ',7P 3 ), . . . , 7r^] where k > 4 and r is simple. 

In the next section we study 2-stack sorting and 2-stack pushall sorting and show the close 
correlation between these two models. This combinatorial study concludes on some partial 
characterization of both classes in terms of permutations they contain or permutations in the 
basis. The key idea is to use the block-decomposition of permutations given in the above 
theorem. 

Then in section [3] we prove that 2-stack pushall sorting can be expressed as a 2-color 
problem on the diagram of permutations. Moreover we characterize diagram of permutations 
that can be colored. This characterization leads to a polynomial algorithm to check whether 
a permutation is 2-stack pushall sortable by finding all colorings for its diagram. 

Section |4] refines the results of section [3] by limiting the number of colorings to test. This 
leads to an optimal algorithm computing in quadratic time a linear representation of all 
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pushall sortings of a given permutation, which thus decides whether a permutation is 2-stack 
pushall sortable. 

To conclude, we give in section some natural continuations of our work. 

2 2-stack sorting vs 2-stack pushall sorting 

In this section we define pushall sorting and point out the close link between 2-stack sorting 
and 2-stack pushall sorting. Moreover, for each of these sorting problems we exhibit some 
recursive necessary and sufficient conditions for a permutation to be sortable depending on 
the root of its block-decomposition. 

In 2-stack sorting, three different operations are allowed as pictured in Figure [2j Each of 
this operation can be encoded with a letter (see Figure [2]). For example, whenever an element 
is popped from stack H and pushed in stack V, we write A. A sequence of operations is 
encoded by a word whose length is the number of operations performed. 

A* A P 
OUTPUT < , , - INPUT 



V H 
Figure 2: Sorting with two stacks in series 

Definition 2.1. A stack word w is a word over the alphabet {p,X,p} such that \w\ p = \w\\ = 
\w\n and for all prefix v of w, \w\ p > \w\\ > \w\^. Intuitively it's a word which describes 
a sequence of appropriate stack operations which take a permutation through two stacks in 
series (without necessarily sorting it). A permutation a = a\ . . . a n is 2-stack sortable if and 
only if there exists a stack word of length 3n (n times each letter p,X,p) which leads to the 
identity in the output with a as input. Such a word is called a valid stack word for a. 

There are several valid stack word for a given permutation: for example, permutation 
2341 admits either ppppXpXXXppp or pppXXXpXpppp as valid words. Note also that p and 
p commutes: if w is a valid stack word for a an w' is obtained for w by exchanging adjacent 
letters p and p, then w' is a valid stack word for a. In his Phd [10] , Murphy studied 2- 
stack sorting by studying stack words. This presentation of 2-stack sorting allow us to define 
formally 2-stack pushall sorting. 

Definition 2.2. A pushall stack word is a stack word such that the first occurrence of letter p 
is after the last occurence of the letter p. A permutation a of size n is 2-stack pushall sortable 
if and only it admits a valid pushall stack word. 

More informally, 2-stack pushall sortable permutations are those which can be sorted by 
pushing all elements in the stacks before writing any element to the output. 

For example 2431 is 2-stack pushall sortable as the word ppppXpXXXppp respects the 
required condition (as does pppXXXpXpppp). 
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Remark 2.3. A stack word w is a pushall stack word if and only if it can be written as 
w = uv with u £ {p, A}* and v £ {A,/i}*. This decomposition is not unique. In the preceding 
example, the word w = ppppXpXXXppp admits two decompositions: w = (pppp)(XpXXXppp) 
and w = (ppppX)(pXXXppp) . 

The previous definition of 2-stack pushall sortable permutations implies that they form 
a subset of 2-stack sortable permutations. Moreover it is easy to check that 2-stack pushall 
sorting is stable by pattern relation: if a is 2-stack pushall sortable then every pattern tt of a 
is 2-stack pushall sortable: choose an occurrence of tt in a and a valid pushall stack word w of 
a. To obtain a valid pushall stack word of tt, delete letters of w that correspond to elements 
of a not involved in the occurrence of tt. The same reasoning holds for general 2-stack sorting. 

Proposition 2.4. 2-stack pushall permutations form a subclass of 2-stack sortable permuta- 
tion class. 

Although we do not know the ratio between these two classes, there exists a close correla- 
tion between them and solving 2-stack pushall sorting is a prerequisite for the more general 
case. We first study the possible configurations of the stacks during a sorting procedure. This 
will help us to obtain properties of stack sorting permutations thanks to their decomposition. 
In a last subsection, we study the basis of 2-stack sortable permutation class and show how 
it is correlated to the 2-stack pushall one. 

2.1 Stack configurations 

At each step of a sorting procedure, some elements of the permutation lie in the stacks. We 
call a stack configuration the position of these elements in stacks H and V . In this section, 
we exhibit a necessary condition on stack configurations to be part of a sorting procedure. 
First we define formally stack configurations. 

Definition 2.5. A stack configuration is a pair of two vectors of positive integers ( V , H) 
of arbitrairy (and maybe different) sizes, such that all coordinates are distincts. A stack 
configuration may be empty (if both vectors are of size zero). Vector V (resp. H) represents 
elements that are in stack V (resp. H) given from bottom to top, so we can apply to stack 
configurations moves X and p, and move p if we know what is the next integer in the input. 

Let a be a permutation, a stack configuration of a is a stack configuration in which coor- 
dinates are bounded by \a\. 

Definition 2.6. To each stack word w of size 3n and permutation a of size n we associate 
a sequence of 3n + 1 stack configurations (ck(w,o~)) describing how the sequence of moves 
w = wi...W3 n take a through the stacks: c\(w,a) is empty and we obtain Ck+i(w,a) from 
Ck(w,a) by doing operation w>k with a as input at the beginning. 

Definition 2.7. Let a be a permutation. A stack configuration c is reachable for a if it exists 
a stack word w and an integer k such that c = Ck(w,a). A stack configuration c is total for 
a if all integers from 1 to \a\ appear in c (this notion depends only on \a\, we don't ask c to 
be reachable for a). 

Remark 2.8. Let w be a stack word of size 3n and a a permutation of size n. Then w is a 
pushall stack word if and only if at least one of the stack configurations (ck{w,a)) is total. 
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During a sorting procedure, stack configurations have constraints so that all elements can 
be popped out in increasing order. Recall that in one-stack sorting, the stack must be in 
decreasing order (from bottom to top). For two-stack sorting, we have the same decreasing 
constraint on stack V but other constraints appear that can be represented as stack patterns. 



Definition 2.9. 

spectively |12| |, 



We call unsortable stack-patterns the following three patterns, denoted re- 
11321 and 121131 - 
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More precisely pattern |12| | means that there is in stack V one element which has a smaller 
element below it. Pattern | |132| means that there is in stack H one element which has a 
greater element below it and a smaller element more below. Pattern |2|13| is somehow special 
as the pattern is divided in both stacks. It means that there are elements a, b, c such that 
b£V,a,c£H,a<b<c and c is above a in stack H . 

Theorem 2.10. A stack configuration can be popped out in increasing order if and only if it 
avoids each unsortable stack-pattern. 

Proof. Notice that if a stack configuration contains any of the 3 unsortable stack-patterns, 
then elements involved in the pattern cannot be popped out in increasing order. 

For the converse, we prove by induction on the number of elements in the stacks that a 
configuration which avoids the 3 unsortable stack-patterns can be popped out in increasing 
order. Suppose that the result has been proved for all stack configurations with at most k 
elements. Note that the result is trivially true for k < 2. Let c be a stack configuration with 
k + 1 elements avoiding the 3 unsortable stack-patterns and m the smallest element of this 
configuration. We show that m can be popped out so that the stack configuration of the k 
remaining elements still avoids the 3 unsortable stack-patterns. Without loss of generality 
assume m = 1. 

Suppose that 1 lies in stack V. As c avoids pattern |12| |, V is in decreasing order so 1 
is at the top of it. It can be popped out and there remains k elements still avoiding the 3 
unsortable stack-patterns. Thus they can be all popped out in increasing order by induction. 

Suppose now that 1 lies in stack H. As c avoids pattern | |132| and 1 is the smallest 
element, all elements above 1 are in increasing order (from 1 to top). All these elements 
can be pushed onto stack V so that stack V remains in decreasing order. Indeed as c avoids 
pattern |2|13|, the top of stack V is greater than the top of stack H. When all elements 
greater than 1 and above 1 in stack H are transferred onto stack V, then 1 can be popped 
out both stacks H and V and the remaining configuration still avoids the 3 unsortable stack- 
patterns (as c avoids pattern | |132|, no pattern |2|13| has been created) and we can apply 
the induction. □ 

Remark 2.11. There is at most one way to pop out in increasing order elements from a 
stack configuration. Indeed to pop out we only use moves fx and X, and if we want to pop out 
in increasing order we have to perform move fj, if and only if the smallest element lies in the 
top of V. 
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Data: a a permutation and c a total stack configuration of a. 
Result: True if c can be popped out in increasing order. 

1 i < — 1; 

2 while i < \a\ do 

3 if topiy) = i then 

4 | pop out top{V) from stack V and let i < — % + 1 

5 else 

6 if H is non empty and top(H) < top(V) then 

7 | pop top(H) from stack H and push it into V; 

8 else 

9 | Return false; 

10 end 

11 end 

12 end 

13 Return true; 



Algorithm 1: Pop out in increasing order 

Proposition 2.12. Let c be a total stack configuration of a permutation o. Then Algorithm^ 
applied to c returns true if and only if c can be popped out in increasing order. Moreover 
Algorithm[I\ runs in linear time w.r.t. \a\. 

Proof: At each step, Algorithm [1] performs either a move [i or a move A. As at most |er| 
moves \i and |<r| moves A can be done, it runs in linear time w.r.t. \a\. We conclude using 
Remark EJJ3 □ 



Theorem 12.101 ensures that a stack configuration can be popped out in increasing order. 
Conditions of this theorem must be verified at each step of a sorting procedure. This is 
formalised in the following proposition: 

Proposition 2.13. If w is a valid stack word for the permutation o, then each stack config- 
uration of (ck(w,o~)) avoids the 3 unsortable stack-patterns. 

The converse is not true: let w = (p\fi) n then for all permutation a of size n each stack 
configuration of (ck(w, cm avoids the 3 unsortable stack-patterns (as it has at most one 
element in the stacks). But if a is not the identity, w is not a valid stack word for a. 

For 2-stack pushall sorting, however, it is sufficient to check whether the stack configura- 
tion obtained just after the last element of a has been pushed onto H avoids the 3 unsortable 
stack-patterns. 

Proposition 2.14. A permutation o is 2-stack pushall sortable if and only if there is a way 
to put all its elements in the stacks so that the total stack configuration obtained avoids the 
three unsortable patterns. 

Proof: If a is 2-stack pushall sortable we conclude using Proposition 12.131 and Remark 12.81 
The converse is a consequence of Theorem 12.101 □ 

2.2 Decomposition and stack sorting 

In this part we exhibit conditions for a permutation o to be 2-stack sorted depending on its 
decomposition. 
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0-decomposable permutations 

Proposition 2.15. A permutation a = Q[ir^ , ir^ 2 \ . . . , 7P fe '] is 2-stack sortable if and only 
if every ir^' for i G {1 . . . k — 1} is 2-stack pushall sortable and ir^ is 2-stack sortable. 

Proof. Suppose that a is 2-stack sortable. Let w a be a valid stack word of a. For i £ {1 . . . k}, 
consider the subword w ^ of w a by taking letters corresponding to an element of tt^K This 
word is of size 3\ir^ | and has equal number of occurrences of the letters p, A, fi. Moreover, it 
is a valid stack word for tp*' as the relative order of elements of tt® under the action of w 
will be the same as the action of w a on a. Furthermore, as the element 1 in a belongs to the 
last block n( k \ all elements of ir® are pushed into the stacks before the first pop. Hence 7r® 
is 2-stack pushall sortable. 2-stack sortable permutations form a permutation class, so that 
7p" must be 2-stack sortable. 

Conversely, if every for i G {1 . . . k — 1} is 2-stack pushall sortable and TT^ is 2-stack 
sortable, let w% (1 < i < A; — 1) be a pushall stack word for and be a stack word for 7r( fc ). 
Then each Wi (1 < i < A; — 1) can be written as u^wf where contains no occurrence of fi 
and u>" no occurrence of p. It is easy to check that the word w[w' 2 . . . w' k _ l Wkw'l_ l w'l_ 2 ■ ■ ■ w'{ 
is a valid stack word for a, hence a is 2-stack sortable. □ 

With a similar proof, we have the following result when restricting to 2-stack pushall 
sortable permutations: 

Proposition 2.16. A permutation a = Q^ 1 ^, n^ 2 \ . . . , ir^] is 2-stack pushall sortable if 
and only if every ir^ for i G {1 . . . k} is 2-stack pushall sortable. 

©-decomposable permutations The case where a is ©-decomposable is a bit different 
as each block of the decomposition can be popped out as soon as they are pushed into the 
stacks. So the only condition is given in the following proposition. 

Proposition 2.17. If a = ©[vrW,. . . ,vr( fc )] then a is 2-stack sortable if and only if each tt^ 1 ' 
is 2-stack sortable. 

For 2-stack pushall sortable permutations, ©-decomposable permutations are harder to 
handle. As no element can be popped out before all elements have been pushed, the element 
1 which belongs to the first block must remain in the stacks until every element is pushed. 
This induces several constraints which are proved in the following propositions. All these 
propositions aim at proving Theorem 12.181 which fully characterizes ©-decomposable 2-stack 
pushall sortable permutations. 

Theorem 2.18. Let a be a ffi- decomposable permutation. Then a is 2-stack pushall sortable 
if and only if a avoids 

B + = {132465, 135246, 142536, 142635, 143625, 153624, 213546, 214365, 214635, 215364, 

241365, 314265, 315246, 315426, 351426, 1354627, 1365724, 1436527, 1473526, 1546273, 
1573246, 1624357, 1627354, 1632547, 1632574, 1642573, 1657243, 2465137, 2631547, 
2635147, 3541627, 4621357, 4652137, 5136427, 5162437, 21687435, 54613287} 

The proof proceeds step by step in Propositions 12.191 to 12.261 
Proposition 2.19. Let a be a permutation such that either: 
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• a G Au(132) 

• a G Av(213) 

• a G ®[At;(132),At;(213)] 

• cr g ®[At;(213),Ai;(132)] 
T/ien a is 2-stack pushall sortable. 

Proof. We show that we can put all elements of a in the stacks so that they avoid patterns 
of Theorem 12.101 (pEJ)- In the first case, just push every element in stack H. For the second 
case, we know from Knuth [U] that each permutation avoiding 231 can be sort in increasing 
order with one stack. So each permutation avoiding 213 can be sort in deacreasing order with 
one stack. Hence we can use stack H to push all elements of a in decreasing order onto stack 
V. For the last two cases, we push the elements in corresponding stacks H for Au(132) and 
V for Av(213). In each case, the stack configuration respect conditions of Theorem 12.101 □ 



Note that Proposition 12.191 give a sufficient condition which is not necessary: the permu- 
tation 143652 is 2-stack pushall sortable but do not belong to one of the preceding cases. In 
this proposition, an important role is given to classes Au(213) and At>(132). These indeed 
are exactly the classes of permutations that can be pushall sorted with a stack configuration 
where all elements lie in one single stack (V for Au(213) and H for Av(132)). Thus the 
only difficult case is whenever a permutation contains both pattern 132 and 213. This is 
characterized by the following proposition: 

Proposition 2.20. A permutation a contains both patterns 213 and 132 if and only if it 
contains one of the following patterns: 1324,2143,2413,3142,465213 and 546132. 



Figure 3: Minimal permutations containing patterns 132 and 213. 



Proof. Minimal permutations that contain both 132 and 213 are exactly permutations of the 
basis of Ad(132) (J Au(213). By minimality of the elements of the basis those permutations 
are at most of size 6 and a comprehensive study ends the proof. □ 

To prove a complete characterization of ©-decomposable 2-stack pushall sortable permu- 
tations, we deal first with permutations whose decomposition contains non-trivial block -i.e. 
blocks not reduced to a singleton-. 

Proposition 2.21. Suppose a = ffifaj . . . a r ] with r > 2, each a% ® -indecomposable and blocks 
ct\ and a r are non-trivial. Then a is 2-stack pushall sortable if and only if o avoids every 
pattern of B\ = {132465,213546,214365,214635,215364,241365,314265,1657243,4652137, 
21687435,54613287}. 
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Proof. We state by checking each pushall stack word of the right size that permutations 
of B\ are not 2-stack pushall sortable. Hence if a is 2-stack pushall sortable it avoids B\. 
Conversely, let a be a permutation avoiding every pattern of B\. As ot\ and a r are non-trivial 
and ©-indecomposable, they contain 21 as a pattern. But a avoids 214365 so that blocks a, 
with 2 < i < r — 1 are trivial. Let / = {i \ contains pattern 132} and J = {j \ aj contains 
pattern 213}. These sets are included in {l,r} and not equal to {l,r} as a avoids 132465 
and 213546. 

• If I = J = 0, then qi G Av(132) and ffi[a 2 . . . a r ] G Av(213) so cr G ffi[Au(132), Av(213)] 
and a is 2-stack pushall sortable by Proposition 12.191 

• If / = and J = {j }, then jo € {l,r}. If jo = 1 then cti G Av(132) and 
ffi[a 2 . . -a r ] G t4i;(213) hence a G ®[Av(132), Av(213)] and u is 2-stack pushall sortable 
by Proposition 12.191 If jo = r, as a avoids 213546 then r = 2, but ct\ G Av (213) 
and a r G ^4t>(132) hence a G ®[Au(213), Au(132)]. So a is 2-stack pushall sortable by 
Proposition 12.191 

• If I = {io} and J = 0, then io G {1,^}- If io = 1, as a avoids 132465 then r = 2, but 
ai G Au(213) and a r G ^w(132) hence a G ®L4u(213), Av(l32)\. So a is 2-stack pushall 
sortable by Proposition 12.191 If io = r then ot\ G Av{132) and ©[a 2 . . . a r ] G ^4f(213) 
hence a G ©L4t> (132), (213)] and a is 2-stack pushall sortable by Proposition 12.191 

• If I = {io} ^ J = {jo}- If io = 1 then jo = r and r = 2 as a avoids 132465. 
But a x G Av(2\3) and a r G Av(132) hence a G ©[Au(213), Au(132)] and a is 2- 
stack pushall sortable by Proposition 12.191 If io = r then jo = 1, a\ G ^4f(132) and 
©[a 2 ...a r ] G Au(213) hence a G ®[Au(132), Au(213)]. So a is 2-stack pushall sortable 
by Proposition 12.191 

• If I = J = {i }, then by Proposition E^Ql a i(1 contains either 1324,2143,2413,3142, 
465213 or 546132. We prove that a contains a pattern of B\. If ctj contains 1324, either 
io < r, and a would contain 132465 or iq = r, and a would contain 213546. Similarly 
if «i contains 2143, a would contain 214365. The same goes for «j containing 2413, 
3142, 465213 or 546132. Hence the case I = J = {io} cannot occur. 

□ 

Given two permutation classes C and C , their horizontal juxtaposition [C C] consists of 
all permutations a that can be written as a concatenation [tt, r] where n is order isomorphic 
to a permutation in C and r is order-isomorphic to a permutation in C . In other words, a 
diagram of a permutation a G [C C] can be divided by a vertical line into two parts, such 
that the left one is order-isomorphic to a permutation of C and the right one to a permutation 

T C 

of C. We can similarly define the vertical juxtaposition 



having a diagram cut by a horizontal line. 



C 



consisting of permutations 



Proposition 2.22. A permutation ©[1, a] is 2-stack pushall sortable if and only if 

o~ G [At>(213) ^4t>(132)] and there exists an associated decomposition a = [n, t] such that there 

are no pattern 213 in a where 2 is in tt and 13 is in r. 
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Proof. If a = [tt, t] with this decomposition satisfying hypothesis of the proposition, then 
®[1,(t] is 2-stack pushall sortable using the following algorithm. Put 1 in H. Then push 
elements of tt in stack V in decreasing order. Then put 1 at top of V and finally push every 
element of r onto H. As there are no pattern 213 in a with 2 in tt and 13 in r, the stack 
configuration respects conditions of Theorem 12.101 hence can be popped out. 

Conversely, suppose that ®[1,<t] is 2-stack pushall sortable and consider a stack word for 
this permutation. As l is the first element, it is pushed at the bottom of H. Then some 
elements are pushed onto l and into V before l is popped out from stack H to stack V. The 
remaining elements are pushed into H as they are greater than 1. We consider the moment 
where all elements have been pushed and l is at the top of V. This separates in two parts 
the elements of a taking r as the elements in H and tt the elements in V apart from 1. From 
Theorem 12.101 decomposition a = [vr,r] satisfies conditions of the statement. □ 

Proposition 2.23. Let E = {<t|©[1,<7] is 2-stack pushall sortable }. Then E is a finitely based 
permutation class whose basis is B 2 = {21354, 24135, 31425, 31524, 32514, 42513, 243516, 254613, 
325416, 362415, 435162, 462135, 513246, 516243, 521436, 521463, 531462, 546132, 4652137}. 

Proof. As 2-stack pushall sortable permutations is a permutation class, so does E. Let B 2 be 
the basis of E. To prove that B 2 is finite, we first prove that every permutation in B 2 has 
size less than 9. Then an comprehensive computation gives the permutations in B 2 . 

By Proposition 12.22] E = {a = ttt \ tt € Av(213),t 6 Av(132) and there are no pattern 
213 in a where 2 is in tt and 13 is in r}. Let a € B 2 . By definition a €" E so a Av(213) and 
o~ Av (132). Let OiOjOk be a pattern 132 such that % is maximal and a r a s at be a pattern 
213 such that t is minimal, then r minimal (for t fixed) and finally s maximal (for t and r 
fixed) . 

• If t < i then tt = a r o- s o~to~iO~jO~k £ E, hence by minimality of the basis a = tt so \o~\ = 6. 

• If t = i then tt = a r a s aiGjak ^ E and by minimality a = tt so \a\ = 5. 

• If t > i, consider the pattern a r a s at (shown in Figure H|). Minimality conditions for t 
and r and maximality condition for s imply that gray zones in the diagram of a are 
empty. So s = t — 1. As a £ E, there is no possible cut a = ttt such that tt € Av(213), 
t G Av(132) and there are no pattern 213 in a where 2 is in tt and 13 is in r. Hence, 
all cuts in a are forbidden, either because they are to the left of a 132 pattern or to the 
right of a 213 pattern or between element 2 and 1 of a pattern 213. More specially the 
cut between t — 1 and t is forbidden. This cut cannot be to the left of a pattern 132 by 
maximality of i (t > i) and cannot be to the right of a pattern 213 by minimality of t. 
So this cut is between elements 2 and 1 of a pattern 213. We consider a pattern 213 
denoted by o x a y a z such that x is minimal and y is minimal for x fixed among patterns 
213 such that x < s = t — 1 and y >t. 
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Figure 4: a r a s a t Figure 5: Cas r > i Figure 6: o x a r o s oto z Figure 7: a r a s ata y a z 

— If r < i then ir = {(JrOs&t&i&j&k&x&y&z} ^ E. Indeed all cuts are forbidden: those 
before r by OiOjOk, between r and s by a r a s at, between s and t by a x a y a z and 
before t by a r a s at- So by minimality of the basis |<r| < 9. 

— If r > i, we want to prove that x < i. Then ir = {o r a s o-to~iO-jO-kO~ x o~yO- z \ ^ E since 
all cuts are forbidden as before and \a\ < 9. As r > i and i maximal, gray zones 
added in Figure are empty. As y > t, a y and a z lie either both in A, or both in 
B, or (T y lies in B and a z in A 

* If <Ty and cj 2 lie both in B, then cr^ lies in D and a x at(y z form the permutation 
132 and as i is maximal, x < i. 

* If cry and (T 2 lie both in A, then a x lies in C and by minimality of y we have 
y = t. x is minimal, so that gray zones added in Figure [6] are empty. Suppose 
that x > i. The cut between i and i + 1 is forbidden as a ^ E. As i is 
maximal the cut cannot be to the left of a pattern 132, neither to the right of 
a pattern 213 by minimality of t. Hence the cut lies between element 2 and 1 
of a pattern 213. Let cr a ai,a c be such a pattern 213 such that a < i and b > i. 
Then a < x and a a lies in area 7 or 5 and c > t by minimality of t. If <r a lies 
in 7 then a a crta c is the pattern 213, which is forbidden by minimality of x. 
Hence a a lies in 5 and b > t otherwise (J a o'bO't is a pattern 213 with a < i < r, 
which is also forbidden by minimality of r. Hence cr a c r b cr c is a pattern 213 with 
a < i < x < s and b > t which is impossible by minimality of x. 

* If a y lies in B and a z in A, by minimality of x, x = r or a x lies in C or cr^ 
lies in D. If cr^ lies in C then o x oto z is a pattern 213 which contradicts the 
minimality of y. If a x lies in L>, a x a r a s is a pattern 132 hence x < i. If x = r, 
by minimality of x then y, gray zones added in Figure [7] are empty. The cut 
between i and i + 1 is forbidden as d ^ E. As before the cut lies between 
elements 2 and 1 of a pattern 213. Let cr a ab(J c such a pattern 213 such that 
a < i and b > i. Then a < r and a a lies in 7 or 5 and c > t by minimality of t. 
If a a lies in 7 then a a atcr c is a pattern 213 and by minimality of x, x < a < i. 
If a a lies in 5 then b > t otherwise Gb<Jt&y is a pattern 132 with b > i, which 
is forbidden by maximality of i. But o a o\,o c is a pattern 213 with a < i and 
b >t, so by minimality of x, x < i. 

□ 



Proposition 2.24. A permutation ©[a, 1] is 2-stack pushall sortable if and only if a € 



Av(132) 
Av(213) 



and there exists an associated decomposition a 



IT 
T 



such that there is no 



pattern 132 in a where element 3 is in 7r and elements 1 and 2 are in r. 
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Proof. Let n = \o~\ + 1. Consider a pushall sorting of ffi[c, 1]. This permutation has n as last 
element, so that we consider the configuration of the stacks just after the insertion of n. By 
Theorem 12. 101 it must avoid the pattern |2|13|, so that all elements in H -under n- are greater 

TT 



than those of V . Hence we can write a 



in H -except n-. Then from Theorem 12.10 



where r contains elements of V and tt those 

T 

tt G Av{12>2) and r G Av (213) and that there are 



no pattern 132 in a where element 3 is in tt and elements 1 and 2 are in r. 

TT 



Conversely, suppose that there exists a decomposition a 



respecting the previous 



conditions then we have a pushall sorting of the permutation ©[cr, 1] using the following 
algorithm. While the input is not empty, if stack H is empty or if the top of H belongs to tt, 
we push the next element of the input onto H. If <7j, the top of H belongs to r, and if the next 
element of the input Uj belongs to r and is greater than <7j, we push o~j onto H, otherwise 
we pop <7j from H and push it onto V. At each step we verify conditions of Theorem 12.101 so 
that all elements can be popped out in increasing order at the end. □ 

Proposition 2.25. Let F = {a\ © [a, 1] is 2-stack pushall sortable }. F is a finitely based per- 
mutation class whose basis is B 3 = {13524, 14253, 21354, 31524, 31542, 35142, 135462, 143652, 
162435, 163254, 246513, 263154, 263514, 354162, 462135, 465213, 513642, 516243, 1657243}. 

Proof. As the set of 2-stack pushall sortable permutations is a permutation class, so is F. By 

Av(132) 



Proposition [2T241 F = {a G 



such that there exists an associated decomposition 



a 



Av(2lS) 

such that there is no pattern 132 in a where element 3 is in tt and elements 1 and 



7T 

r 

2 are in r}. Hence E and F are in one-to-one correspondence by taking an element of E, 
rotate its diagram by —tt/2 and apply the symmetry with respect to axis (Oy). If elements 
are in one-to-one correspondence by rotation and symmetry so does the basis which proves 
the result. □ 

Proposition 2.26. A permutation ©[1,<t, 1] is 2-stack pushall sortable if and only if a G 
®[At;(213),At;(132)]. 

Proof. By Proposition 12.221 ®[1,<t, 1] is 2-stack pushall sortable if and only if ®[<r, 1] G 
[^4u(213), -Av(132)] and there exists a corresponding decomposition a = ttt such that there 
is no pattern 213 in a where element 2 is in n and 13 are in r, which is equivalent to 
a G L4.-u(213), Av (132)1 and there exists a corresponding decomposition a = ttt such that 
there are no pattern 21 in a where element 2 is in tt and element 1 is in r, i.e. a G 
®\Av{2\$),Av{Yi2)\. □ 

We are now able to prove Theorem 12.181 (pEJ). 

Proof. Permutations of B+ are not 2-stack pushall sortable (check each pushall stack word 
of the right size), hence if a is 2-stack pushall sortable it avoids B + . Conversely suppose 
that a avoids B + . Let a = ®[a\.. .a r ] be the ©-decomposition of a with r > 2 and ctj 
©-indecomposable for all i. 

• If oi\ and a r are non trivial then a is 2-stack pushall sortable thanks to Proposition 12.21] 
Indeed a avoids B x = {132465, 213546, 214365, 214635, 215364, 241365, 314265, 1657243, 
4652137,21687435,54613287} as B 1 C B+. 
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• If ax is trivial then a = ©[1, vr] and tt avoids B 2 = {21354, 24135, 31425, 31524, 32514, 
42513, 243516, 254613, 325416, 362415, 435162, 462135, 513246, 516243, 521436, 521463, 
531462,546132,4652137} so that a is 2-stack pushall sortable by Proposition 

• If a r is trivial then a = ©[vr, 1] and vr avoids B 3 = {13524, 14253, 21354, 31524, 31542, 
35142, 135462, 143652, 162435, 163254, 246513, 263154, 263514, 354162, 462135, 465213, 
513642,516243, 1657243} hence a is 2-stack pushall sortable by Proposition E25J 

□ 

We call separable permutations the class Au(2413, 3142). 

Theorem 2.27. Let a be a separable permutation, a is 2-stack pushall sortable if and only if a 
avoids B = {132465, 213546, 214365, 1354627, 1436527, 1624357, 1632547, 1657243, 4652137, 
21687435,54613287}. 

Proof. As permutations of B are not 2-stack pushall sortable, every 2-stack pushall sortable 
permutation avoids B. Conversely, supppose that a avoids B. As a is separable, a is either ffi- 
decomposable or ©-decomposable or trivial (i.e. of size 1), and a avoids 2413 and 3142 which 
added to constraints of B gives that a avoids B + , the set defined in Theorem 12.181 If a is 
©-decomposable, then a is 2-stack pushall sortable by Theorem l2.18[ If a is ©-decomposable, 
then a = ©[vr* 1 ), ir( 2 \ . . . ,7r( fc )] where each 7rW is either trivial or ©-decomposable. So a is 
2-stack pushall sortable by Proposition 12.161 and Theorem 12.181 □ 

2.3 Basis of stack sorting class 

In the previous section, we show that 2-stack pushall sortable separable permutations form 
a finitely based permutation class. This property does not hold for 2-stack pushall sortable 
permutations and we exhibit an infinite antichain in the following proposition. 

Proposition 2.28. The basis of 2-stack pushall sortable permutation is infinite. 

Proof. Consider permutations 2n — 3 2n — 1 2n — 5 2n . . .p p + 5 . . .1 6 2 4 for n > 3. The 
first ones are depicted in Figure El These permutations are simple and incomparable. To 
complete the proof, straightforward though technical, just check that those permutations are 
not 2-stack pushall sortable and that every pattern of these permutations are 2-stack pushall 
sortable. 



Figure 8: An antichain of the basis of 2-stack pushall sortable permutations class. 

□ 

Note that the basis is infinite and contains a infinite number of simple permutations, and 
the 2-stack pushall sortable class contains also an infinite number of simple permutations. 
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Proposition 2.29. If a is in the basis of 2-stack pushall sortable permutations, then a is 
2-stack sortable. 

Proof. Let a = a\Ui . . . a n be in the basis of 2-stack pushall sortable permutations. By 
definition, o\02 ■ ■ • o~ n -i is 2-stack pushall sortable. We can sort a (not pushall sort a) using 
the following algorithm. Push all elements o\ to cr n _i in the stacks following the 2-stack 
pushall sortable operations of a\ . . . a n -\- Then pop elements 1, 2, . . . , a n — 1, then push a n 
and pop it to the output and pop the remaining elements. It is easy to check that these 
operations are allowed. □ 

Those last two propositions give a partial characterization of the basis of 2-stack pushall 
sortable permutations class and 2-stack sortable permutations. A more accurate result can 
be given for certain type of permutations in the basis. 

Proposition 2.30. Let ir be a 0- decomposable permutation. Then ir belongs to the basis of 
2-stack sortable permutations class if and only if tt = Q[a, 1] where a belongs to the basis of 
2-stack pushall sortable permutations class. 

Proof. Let tt = Q[a, 1] with a a pemrutation of the basis of 2-stack pushall sortable permu- 
tations class. Proposition 12.151 ensures that tt is not 2-stack sortable. Note also that every 
pattern of tt is 2-stack sortable. To prove this result, suppose that you remove a point in the 
permutation. Suppose we delete element 1 then the obtained permutation is a, hence it is 
2-stack sortable by Proposition 12.291 Otherwise we delete an element of a leading to a' which 
is 2-stack pushall sortable by the definition of a permutation class basis. Then, Q[a', 1] is 
2-stack pushall sortable using Proposition 12.151 

Conversely, if a = 0[7tW,7t( 2 ), . . . ,tt^] belongs to the basis of 2-stack sortable permuta- 
tions class, then by Proposition [2J2J either ir^ is not 2-stack sortable which contradicts the 
minimality of a {a is an element of the basis so that every pattern of a must belong to the 
class) or there exists 1 < i < k — 1 such that 7rW is not 2-stack pushall sortable. But in that 
case, G[tt^\ 1] is not 2-stack sortable by Proposition 12.151 hence a = Q[^ % \ 1] by minimality 
of basis elements. If tt® has a proper pattern r which is not 2-stack pushall sortable then 
Q[t, 1] is a proper pattern of a which is not 2-stack sortable. This is impossible as a belongs 
to the basis of 2-stack sortable permutations class. So belongs to the basis of 2-stack 
pushall sortable permutations class, which concludes the proof. □ 



3 Sorting and bi-coloring 

3.1 A simple characterization 

There is a natural relation between 2-stack pushall sorting and coloring of permutation dia- 
gram into two colors. The key idea is to look at the stack configuration once all elements of 
the permutation are pushed into the stacks. Then each element of the permutation belong 
either to stack H or to stack V. We assign a color to them depending in which stack they lie 

at this particular step of the sorting. In this article we color like points that lie in stack 



H and like I I points in stack V . 



However by Remark 12 .31 this stack configuration is not unique, and neither is the coloring. 
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Definition 3.1. A bicoloring of a permutation a is a coloring of the points of the diagram 
of a with two colors G and R. 

A valid coloring is a bicoloring which avoids each of the four following colored pattern: 

• pattern 132: there is a pattern 132 in R 

• pattern 213: there is a pattern 213 in G 

• pattern 1X2: there is a point of R lying vertically between a pattern 12 of G 

• pattern 2/13: there is a point of G lying horizontally between a pattern 12 of R 

Definition 3.2. Let a be a permutation. To each total stack configuration of a the map Col 

assigns the bicoloring of a such that elements of H are in R and elements of V are in G. 
To every bicoloring of a permutation a the map Conf associates the total stack configuration 
of a such that elements of G lie in V in decreasing order of value from bottom to top and 
elements of R lie in H in increasing order of indices from bottom to top. 

Remark 3.3. For any bicoloring b, Col(Conf(b)) = b. For any stack configuration c such 
that elements of V are in decreasing order of value from bottom to top and elements of H are 
in increasing order of indices from bottom to top, Conf (Col (c)) = c. 

Proposition 3.4. Let b be a bicoloring of a permutation a. Then Algorithm^ applied to b 
returns true if and only if Conf (b) is reachable for a. In this case the stack configuration to 
which Algorithm^ leads is Conf(b). 

To state this proposition we need the two following lemmas: 

Lemma 3.5. At each step of Algorithm the stack configuration we have is reachable for 
a , elements of H are in increasing order of indices from bottom to top, elements of V are 
in decreasing order of value from bottom to top, there is no element of R in V , there is no 
element of R above an element of G in H and elements of G that lie in H are in increasing 
order of value from bottom to top. 

Moreover index i verifies that if i < \o~\ then Oi is the next element of the input and if 
i > \o~\ then there is no more element in the input. 

Proof: The proof is by induction on the number of stack operations performed by the algo- 
rithm. Algorithm [2] begins with the empty stack configuration and o as input and i = 1 so 
the properties are true at the beginning. Algorithm [2] performs only appropriate stack oper- 
ations so at each step the configuration obtained is reachable for o~. Moreover in a reachable 
configuration, elements of H are in increasing order of indices. When an element is put in V 
(this happens at line 9 or 21) then this element is in G (checked at line 6 or 19) and is smaller 
than the top of V (checked at line 8 or 20) so that elements of V remain in decreasing order 
of value from bottom to top and V contains no element of R. When we put an element in 
H, it can be at line 4 or 14. In the first case, H is empty or its top is in R (checked at line 
3) so all its elements are in R by induction hypothesis. In the second case, the top of H is 
in G and the element we put in H is in G and greater than the top of H. This ensures that 
there is no element of R above an element of G in H and that elements of G that lie in H are 
in increasing order from bottom to top (using induction hypothesis). Finally i is increased 
exactly when <jj is put into H so the last property remains true. □ 
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Data: a a permutation and b a bicoloring of a. 

Result: True if the stack configuration corresponding to b is reachable from a. 

1 Begin with the empty stack configuration and a as input and i = 1; 

2 while i < jcr do 

3 if H is empty or top(H) G f? then 

4 push <jj into If; 

5 i « — i + 

6 else /* top(H) G G */ 

7 if <jj G -R or Oi < top(H) then 

8 if V is empty or top(H) < top(V) then 

9 | pop top(H) from stack H and push it into V; 
10 else 

n [ Return false; 

12 end 

13 else /* top(H) G G ; Oi G G and Oi > top(H) */ 

14 push o"j into H ; 

15 i < — i + 1; 

16 end 

17 end 
is end 

19 while H is nonempty and top(H) G G do 

20 if top(H) < top(V) then 

21 j pop top(H) from stack H and push it into V; 

22 else 

23 [ Return false; 

24 end 

25 end 

26 Return true; 



Algorithm 2: Algorithm to obtain a reachable configuration compatible with a bicol- 
oring 
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Lemma 3.6. Algorithm^ terminates in linear time w.r.t \a\. 

Proof: At each step, Algorithm [5] performs either a legal move p, or a legal move A, or return 
false or true (and stops). As at most |<r| legal moves p and |<r| legal moves A can be done, 
Algorithm [2] terminates after at most 2\o~\ + 1 steps. As each step is done in constant time, 
we have the result. □ 

We are now able to prove Proposition I3.4t 
Proof: If Algorithm [2] applied to b returns true, then it reaches line 26. In particular the loop 
of line 19 stops so the top of H is not in G. Thus by Lemma 13.51 there is no element of G 
in H. In addition by the same lemma elements of H are in increasing order of indices from 
bottom to top, elements of V are in decreasing order of value from bottom to top and there is 
no element of R in V. So the stack configuration we have is Con f {Jo). Moreover Lemma 13.51 
states that the stack configuration we have is reachable for a, so Conf(b) is reachable for a. 

Conversely if Conf(b) is reachable for a, then there is a sequence w of appropriate stack 
operations so that the configuration obtained with a as input is Conf(b). Let us prove that 
the sequence of moves w' performed by Algorithm [2] applied to b is w. We prove by induction 
on k < \w\ (k > 0) that w and w' have the same prefix of length k (obvious for k = 0). 
First notice that as Conf(b) is a total stack configuration, so w has no letter p, and that 
Algorithm [2] performs only moves A and p, so w' has no letter p. Suppose that w and w' have 
the same prefix v of length k with k < \w\, let d be the stack configuration obtained after 
permforming moves of v with a as input. We want to prove that w' k+l exists and w' k+l = Wk+i- 
By definition of w' , w' k+l is the move performed by Algorithm [2] in configuration d (setting 
by extension w' k+l = a if Algorithm [2] terminates in configuration d, i.e. if \w'\ = k), and 
by defintion of w, Wk+i is a move which allows to go from configuration d to configuration 
Conf(b) (maybe with some additional moves). 

We check the value of i after Algorithm [2] has performed moves v. We know that at this 
step stacks are in configuration d. 

If i > \a\, then from Lemma 13.51 in configuration d all elements of a lie already in the 
stacks. As w is a sequence of appropriate stack operations, then w^+i ^ p so w^+i = A (w 
has no letter p). As wt+i is a move which allows to go from configuration d to configuration 
Conf(b) in which there is no elements of R in V and V is decreasing, then the top of H in d 
is in G and smaller than the top of V (or V is empty). As i > \a\ and as the top of H in d 
is in G and smaller than the top of V (or V is empty) then Algorithm [2] performs line 21 so 

w k+l = A = w k+l- 

If i < \o~\ then we are in the loop beginning at line 2 of the algorithm and from Lemma l3.5l 
cij is the next element of the input. Suppose that w^+i = A. As w^+i is a legal move which 
allows to go from configuration d to configuration Conf{b) in which there is no elements of 
R in V and V is decreasing, then H is non empty, the top of H is in G and smaller than the 
top of V. Suppose in addition that (Tj G G. As a, is still on the input after Wk+i and Wk+\ 
is a move which allows to go to configuration Conf{b) in which V is decreasing, then a is 
smaller than the top of H in d. So either cjj < top(H) or crj € R. So from d Algorithm [2] 
performs line 9 so w' k+l = A = w^+i- 

Suppose that w^+i = p- If in configuration d stack H is empty or top(H) £ R then 
Algorithm [2] performs line 4 so vj' k+1 = p = Wk+i- Otherwise let ah be the top of H in d, then 
ah £ G. So ah £ V in Conf(b). But once Wk+i = p is performed cij is above ah in H. As 
Wk+i is a move which allows to go from configuration d to configuration Conf(b) then cij is 
below ah in V in Conf(b) (indeed it is impossible that ah goes to stack V and Uj remains in 
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stack H). So <Tj G G and as in Conf(b) elements of V are in decreasing order, Oi> o^. So the 
test of line 7 of the algorithm is false and Algorithm [2] performs line 14 so w' k+l = p = wt+i- 
This ends the induction. We have proved that w is a prefix of w' , so Algorithm reaches 
configuration Conf(b). We have now to prove that Algorithm [2] stops in this configuration 
and returns true. 

When Conf(b) is reached then there is no element in the input anymore, so from Lemma [3.5l 
i > \a\, and top(H) ^ G in Conf(b). So both loops while of Algorithm [2] are finished and the 
algorithm reaches line 27, returns true and terminates in configuration Conf(b). □ 

Lemma 3.7. Let b be a bicoloring of a permutation a. If Algorithm^ applied to b returns 
false then b has a pattern 1X2 or a pattern 213. 

Proof: We consider the stack configuration reached when Algorithm [2] returns false. We 
set ah = top(H) and a v = top(V). By Lemma I3T51 a v G G. Algorithm [2] returns false by 
reaching either line 11 or line 23. In both cases, ah G G and ah > a v . Now we consider the 
step of the algorithm where a v was put in V, the indice % at this step of the algorithm, and 
the corresponding configuration c just before the move putting a v into V is done. So at this 
step a v is on the top of H, and i > v. If ah is in H in c, then it is below a v , contradicting 
Lemma 13.51 (ah > cr v and both are in G). As ah is in H when the algorithm ends, it cannot 
be in V in c. So ah is still in the input and i < h < \a\. Recall that we consider the step of 
the algorithm where a v is put in V . This can happen at line 9 or 21 but i < \a\ so it is at 
line 9. So the test of line 8 is true, thus either a% G R and then a v , ai, ah is a pattern 1A2 of 
b, or <jj G G but <j.; < a v and then a v ,ai, ah is a pattern 213 of b. □ 

Theorem 3.8. The map Col is a bisection from the set of reachable total stack configuration 
of a avoiding the three unsortable patterns to the set of valid coloring of a. Moreover the 
inverse of Col is the map Conf. 

Proof: Let c be a reachable total stack configuration of a avoiding the three unsortable 
patterns and set b = Col(c). We have to prove that b avoids every forbidden colored pattern 
of Definition 13.11 

If b has a pattern 132 in R then there are three element Uj, Oj and a^ of R such that 
i < j < k and ai < a^ < aj. By definition of Col, ai, aj and a^ lie in H. As c is reachable 
and i < j < k, ai is below aj which is below a^. So we have a stack-pattern | |132| in c which 
contradicts our hypothesis. So b has no pattern 132. 

If b has a pattern 213 in G then there are three element <7j, aj and a^ of G such that 
i < j < k and aj < ai < <7&. By definition of Col, ai, aj and a^ lie in y. As c avoids 
stack-pattern |12| |, a^ is below cij which is below aj. But then c is not reachable: as a^ is 
below (Tj and aj in V, ai and aj have to stay in stack H until a^ enters stack H. But as i < j, 
ai is below aj in stack H and cannot be below aj in stack V as going from stack H to stack 
V reverse the order. So b has no pattern 213. 

If b has a point of R lying vertically between a pattern 12 of G then there are elements 
ai and aj of G and a^ of R such that i < k < j and < <7j. By definition of Col, ai and 
aj lie in V and a^ lies in i7. Configuration c is reachable. We consider a sequence of stack 
operations leading to c. As i < k, ai is already in the stacks when a^ enters H. As a^ remains 
in H in c but Uj is in V in c, <7j has to be already in V when cr^ enters stack H. As k < j, at 
this moment aj is not already in stack V, so cij will be above ai in V and they form a pattern 
|12| | in c, which is excluded. So b has no pattern 1X2. 
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If b has a point of G lying horizontally between a pattern 12 of R then in c these points 
form a pattern |2|13| which is excluded. So 6 has no pattern 2/13. 

Conversely let b be a valid coloring of a. By definition Conf(b) is a total stack configura- 
tion of a. We have to prove that ConfifS) is reachable for a and avoids the three unsortable 
stack patterns. As b is a valid coloring, it avoids patterns 1X2 and 213. So from Lemma 13.71 
Algorithm [2] started with input b returns true. Thus from Proposition 13. 4[ c is reachable for 
a. Moreover by definition of Conf , Conf(b) avoids pattern |12| |. Furthermore we know 
that in Conf(b), elements of H are in increasing order of indices from bottom to top. So if 
Conf(b) has a pattern | |132|, then b has a pattern 132, and if Conf(b) has a pattern |2|13| 
then b has a pattern 2/13. A b is a valid coloring, we conclude that Conf{b) avoids the three 
unsortable stack patterns. 

Now using Remark 13.31 it's clear that Conf is the inverse of Col. □ 

Theorem 3.9. A permutation a is 2-stack pushall sortable if and only if its diagram admits 
a valid coloring. 

Proof: Consequence of Proposition 12.141 and Theorem 13.81 □ 
Now thank to Theorem 13.91 we have a naive algorithm to check if a permutation a is 
2-stack pushall sortable: forall bicoloring b of a, we can test if b is valid by checking if b 
avoids patterns 213, 1A2, 2/13 and 132 of Definition 13.11 But first notice that we have a 
more efficient way to test if a bicoloring is valid: 

Proposition 3.10. Let b be a bicoloring of a permutation a. We can check in linear time 
w.r.t. \a\ ifb is a valid coloring. More precisely, b is a valid coloring if and only if Algorithm^ 
applied to b returns true and Algorithm^]! applied to Conf(b) returns true. 

Proof: From Theorem 13.81 b is valid if and only if Conf(b) is reachable for a and avoids the 
three unsortable patterns. We conclude using Lemma [3.61 Proposition 13. 41 Theorem 12.101 and 
Proposition [27121 □ 

Now even using this efficient way to test if a bicoloring is valid, the naive algorithm 
descrided above is unefficient. Indeed there is 2^ bicolorings of a, leading to a exponential 
algorithm. Yet we will find a way to restrict the possible number of colorings to a polynomial 
number. The key idea is to look at increasing sequences in the permutation. 

3.2 Increasing sequences in a valid coloring 

First we reformulate the notion of valid coloring thanks to increasing and decreasing sequences. 

Proposition 3.11. Let c be a bicoloring of a permutation a. Then c is a valid coloring if 
and only if c respects the following set of rules denoted 1Z$ : 
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Figure 9: Rewriting rules TZs 



For example rule (i) means that if two points (i,o~i) and (j,o~j) are in increasing order 
i < j and o~{ < o~j and belong to G then every point (k,o~k) of the permutation must respect: 

• If i < k < j then o~k > o~i and (k,o~k) belongs to G. 

• If k < i and o~i < < o~j then (k^a^) belongs to R. 

Proof. We prove that c is not valid if and only if c violates a rule of 1Z&. Suppose that c is not 
valid then c has one of the four colored patterns of Definition 13. 11 If c has a pattern 132 then 
c violate rule (ii) applied to elements 1 and 3 of the pattern 132, as element 2 of the pattern 
lies in a zone that should be empty. If c has a pattern 213 then c violate rule (i) applied to 
elements 2 and 3 of the pattern 213, as element 1 of the pattern lies in a zone that should be 
empty. If c has a pattern 1X2 then c violate rule (i) applied to elements of G of the pattern 
1X2. If c has a pattern 2/13 then c violate rule (ii) applied to elements of R of the pattern 
2/13. Conversely if c violates a rule of 7^8 then a comprehensive study shows that c has one 
of the four colored patterns of Definition 13.11 and is not valid. □ 

We can use implication rules of TZs to limit the number of bicoloring to test, using the 
following idea: knowing the coloring of some points in the permutation (either in R or in G), 
the deduction rules given in Figure [9] can be applied until we obtain either a contradiction 
or no more rule can be applied. We can try the following algorithm: Set the color of two 
increasing points of a, use implication rules to deduce the color of the other points and test 
whether the coloring obtained is right. Unfortunately, implication rules are not sufficient to 
ensure that given the color two points, the color of all other points is set. We may have to 
choose arbitrary the color of lots of points. To ensure that the number of bicoloring to test 
is polynomial, we have to study more precisely properties of increasing sequences in a valid 
bicoloring. 

Definition 3.12. Let c be a bicoloring of a permutation a. We call increasing sequence RG 
a pair of two points (ai,aj) such that i < j, o~i < Oj, <7j € R and <jj € G. We define in the 
same way increasing sequences GR, RR or GG. 

Rule (Hi) of 7^-8 implies that every increasing sequence RG fixes the color of all points to 
the left of G{ below o~j (which are in R) and to the right of o~i above Oj (which are in G). The 
following theorem shows that when a is Q-indecomposable, the color of points to the left of 
dj above ga is also fixed. 



21 



Theorem 3.13. Consider a valid coloring of a Q- decomposable permutation a. If there exist 
two points &{ < <jj,i < j such that at G R and tjj G G, then the color of every point ak with 
k < i or at > o-j is determined by iterations of rules Cs knowing only the color of <7j and 
aj and can be represented as follows, the second diagram being a short representation of this 
alternance which will be used in the sequel. Furthermore, any increasing sequence (i, Cj), (j, aj) 
of points located either to the left of i or to the top of aj is either monochromatic or colored 
RG. Moreover, knowing o"j and aj, we can decide the color of the points whose indices are 
less than i or whose values are greater than aj in linear time. 




Figure 10: Zone Arg 



Proof: The proof is by induction on the assigned border between the zone not yet assigned to 
a stack and the assigned zone containing cij and aj where points are forced to be in a specific 

stack. At first, the assigned border is reduced to the segments (i, 1) (h°~j) ( n i a j) as 

well as the assigned zone (where n = \a\). 

More formally we build sequences (<7j fc ) and (aj k ) such that ai k aj k in an increasing sequence 
RG and the color of all points lying in the set Ck = {ae | ik < £ < i and aj < at < aj k } is 
determined and respects Figure [10J We set io = i and jo = j. We prove that if (cr^ / 1 or 
aj k j£ n) then we can build <7; fe+1 and aj k+1 such that ai k+1 < ai k or aj k+1 > a jk . 

We set Hk = {ae \ £ < ik and at < aj k } and = {at \ t > %k and at > aj k } (see 
Figure CD]). By rule (iii) applied to at k and aj k , C H and Vk C V. Then, different 
situations may happen depending on whether areas and Vk are empty: 

Hk and Vk empty: Then a is 0-decomposable which is in contradiction with our hypothesis. 

Hk and Vk both non empty: If both of the colored zones Hk or Vk are non empty, we set 
ik+i = min{£ \ at € Hk} and <x, fc+1 = max Vk (see Figure [TT]) . Then Ck+i = Ck U Hk UV^U Z 
is a partition of Ck+i, where Z = {at \ ik+i < £ < ik and aj k < at < &j k+1 } (see Figure [TT]) . 
The only points of Ck+i whose color is not determined yet are those of Z. If Z is not empty 
consider a point at of Z. If at G V then rule (i) applied to at and aj k+1 is in contradiction 
with the existence of <7j fc . Hence at G H but then rule (ii) applied to ai k+1 and at is in 
contradiction with the existence of aj k . So Z is empty and the color of all points of Ck+i is 
determined and respects Figure [TOT 
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Figure 11: Transitive closure 



Only one area in Hk and is empty: The same proof as the preceding case allow us 
to define a new point Cj fc+1 or 0"j depending on which area is empty and we can extend the 
assigned border as shown in next figure. 
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Transitive closure 

Hence, the assigned zone keeps growing until all permutation points are assigned, proving 
Theorem [3J3J □ 

We also have a similar result extending rule (iv): 

Theorem 3.14. Consider a valid coloring of a Q-decomposable permutation a. If there exist 
two points o~i < o~j,i < j such that <7j £ G and o~j 6 R, then the color of each point o~k with 

I 



k > j or &k < °i determined. Such a zone will be represented as 



in the sequel. 



Proof: Notice that rules are symetric so that the same proof as for Theorem 13. 131 holds. □ 

Knowing Theorem 13.131 and Theorem 13. 141 to set the color of as much points as possible, 
we better have to choose the lower right increasing sequence RG or the upper left increasing 
sequence GR. Let us now define properly these particular ascents. 

We consider a valid bicoloring c of a permutation a. We define Arq as the set of increasing 
sequences RG of c. 

Lemma 3.15. Suppose Aug ^ 0- Among increasing sequences RG of c, the pair (ai,aj) 
which maximizes i first then minimizes aj (for i fixed) is the same than the pair that minimizes 
o~j first then maximizes i (for aj fixed). 

Proof. Let (<7i ,<7j ) be the pair that maximizes io first then minimizes o~j Q and (cr^,^) be 
the pair that minimizes aj 1 first then maximizes i\. Then by definition io > i\ and aj 1 < aj . 

If ji < io then {o~j 1 ,aj Q ) is an increasing sequence GG and rule (i) is in contradiction 
with <7j € H as j\ < io < jo . If aj 1 < then (a^ , <7j ) is an increasing sequence RR 
and rule (ii) is in contradiction with aj 1 G V as er^ < aj 1 < o"j . Hence (<7j , aj 1 ) is an 
increasing sequence RG. Then by definition of jo, o~j < o~j 1 and by definition of i\, i± > io- 
So (o- io ,a jo ) = {o-i^o-j-^). □ 
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By the preceding lemma, when Arc / we can define iRG,3RG as the lower right 
increasing sequence RG. By symmetry, we can also define icRi 3gr the upper left increasing 
sequence GR when Arq ^ 0, where Aqr is the set similar to Arq but for increasing sequences 
GR. 

Now we have all the tools to prove that there are only a polynomial number of bicolorings 
to test. We juste have to do a case study depending on Arc or Aqr are empty. 



3.3 Case study 

Recall that from Proposition 12.161 if a is 0-decomposable then a is 2-stack pushall sortable 
if and only if each 0-indecomposable block of a is 2-stack pushall sortable. Thus, we can 
assume that a is 0-indecomposable. 

In this section, we consider a valid coloring c of a 0-indecomposable permutation a. We 
prove that knowing if there are ascents RG or GR in c and knowing irg, Jrg-, iGR and jgr 
(if they exist), we can deduce the color of every point of o. 

We prove this considering 4 cases depending on whether there are ascents RG or GR in c. 



3.3.1 There is no bicolored ascents 

If Arc and Aqr are both empty, then the coloring is monochromatic: 

Proposition 3.16. Let a be a Q-indecomposable permutation and c a valid coloring of a such 
that every pattern 12 of a is monochromatic. Then all points of a have the same color. 

Proof: Let <7j and o~j be two consecutive left-to-right minima of a. By definition there are 
no point below cij and to the left of o~j as shown by the empty sign in the following figure 

Ik 



?i 

. As a is 0-indecomposable, there exist a point above Uj and to the right of <7j. As 
increasing subsequences are monochromatic, <7j and have the same color. The same goes 
for <7j and a^. Thus o~i and o~j have the same color. So all left-to-right minima of a have the 
same color. By definition of left-to-right minima, for every non-minimal point a\ there exists 
a left-to-right minima a m such that (o~ m , a{) is a pattern 12 of a. Thus 07 has the same color 
as a m , and all points of a have the same color. □ 



3.3.2 There is no increasing sequence RG but some increasing sequences GR 



We suppose in this section that there exists at least one increasing sequence GR but no 
increasing sequence RG. As Aqr is non empty, icR and jgr are defined. We prove that once 
iGR and jcR are determined, then it fixes the color of every other point of the permutation. 

Proposition 3.17. Let a be a Q-indecomposable permutation and c a valid coloring of a such 
that there is no increasing subsequence RG in c and there is at least an increasing sequence 
GR in c. Then c has one of the following shapes (where maybe a = iqr or b = jgr) : 
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Remark 3.18. Here and in all the following, when a zone of a diagram is colored with R 
(resp. G), it means than if there are some points lying in this zone, they are in R (resp. G). 
And when a zone of a diagram has an empty sign, it means than this zone is empty. 

Proof: The color of every point such that k > jgr or cr& < Oi GR is determined by 
Theorem 13.141 (see the first diagram of Figure [12]) . Note that we denote by * the zone 
where the color of the points is unknown. By maximality of o~i GR , any point above o~i GR and 
lower left with respect to o~j GR is in R. By minimality of jgr, an Y point to the left of Cj GR 
and top right with respect to 0"i Gfl . is in G. As no point can be both in R and in G, we know 
that the zone between o~i GR and o~j GR is empty, as shown in the second diagram of Figure [12] 
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Figure 12: Only bicolored increasing subsequences GR exist 



Let a be the leftmost point among points below ioR (notice that a may be equal to (gr)- 
Applying rule (i) to a and icR, we obtain the third diagram (note that if a = icR the column 
between igr and a does not exist). Let b be the topmost point to the right of jgr (b may be 
equal to jgr)- Applying rule (ii) to jgr and b, we obtain the fourth diagram of Figure [T2l (if 
b = jgr the column between jgr and b does not exist). 

At last, we number two different areas and discuss about the different cases whether these 
zones are empty or not. These zones are pictured in the fifth diagram of Figure [T2l 



Zone 1 is not empty Let x be the leftmost point inside zone 1. Note that x may be above 
or below jgr- First diagram of Figure [13] illustrates the position of point x. Applying rule (ii) 
to x and b we obtain the second diagram of Figure [TBI By hypothesis, there are no increasing 
sequence RG, thus there are no points in G in the up-right quadrant of x. This leads to the 
third diagram. At last, if the zone * is not empty, then a is 0-decomposable by cutting along 
the row of b and the column of x. Thus * is empty and all points have a determined color, as 
in the first diagram of Proposition 13.171 



* 


* 







0. 


1 X 


« 


b 

I. 

JGR 





0. 


i •GH 
a 





* 















b 




I X 


< 


IGR 





01 


, l GR 

a 





* 











0' 




« 


b 

i 

JGR 





0. 


. l GR 

a 





» X 


> 


b 

JGR 


0' 


, l GR 

a 





Figure 13: Zone 1 is not empty 
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Zone 1 is empty Suppose that zone 1 is empty. If zone 2 is also empty then as a is Q- 
indecomposable, zone * is also empty and all points have a determined color, as in the second 
diagram of Proposition 13.171 
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Figure 14: Zone 1 is empty 



Otherwise, zone 2 is not empty and let x be the topmost point inside zone 2 (x may be 
to the left or to the right of i GR ). This is depicted in the second diagram of Figure [HJ We 
apply rule (i) to a and x to obtain the third diagram. As there is no increasing subsequence 
RG, there is no point of R in the lower left quadrant of x as depicted in the fourth diagram. 
Moreover, a is Q-indecomposable, thus zone * is empty and each point has a determined 
color, as in the last diagram of Proposition 13.171 □ 

Definition 3.19. Let a be a permutation and i and j two indices of a such that OiOj is 
an ascent. Set a = min{A; | < <Tj} and b such that = max{<Tfc [ k > j}. We define 
CQn{a,i,j) as the partial bicoloring of a having the following shape: 




Proposition 3.20. Let a be a Q-indecomposable permutation and c a valid coloring of a such 
that there is no increasing subsequence RG in c and there is at least an increasing sequence 
GR in c. Then c = C GR (a,i GR ,j GR ). 

Proof: This is a direct consequence of Proposition 13.171 and Definition 13.191 □ 
3.3.3 All bicolored increasing sequences are labeled RG 

We suppose in this section that there exists at least one increasing sequence RG but no 
increasing sequence GR. As A RG is non empty, i RG and j RG are defined. We prove that once 
i RG and j RG are determined, then it fixes the color of every other point of the permutation. 

Proposition 3.21. Let a be a Q-indecomposable permutation and c a valid coloring of a such 
that there is no increasing subsequence GR in c and there is at least an increasing sequence 
RG in c. Then c has one of the following shapes (where maybe a = j RG or b = i RG ): 
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Proof: The color of every point such that A; < irg or cr^. > <7j BG is determined by Theo- 
rem 13.131 (see the first diagram of Figure [15]) . We denote by * the zone where the color of the 
points is unknown. By maximality of irg an d minimality of o~j GR we know the color of some 
other points, and as no point can be both in R and in G, we know that the zone between 
o~i R Q and o-j HG must be empty, as shown in the second diagram of Figure EJ 
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Figure 15: All bicolored increasing sequences are labeled RG 



Let a be the rightmost point among points above jrg (maybe a = jrg)- Rule (i) applied 
to points jrg and a gives the third diagram of Figure [15] (note that if a = jrg the column 
between jrg and a does not exist). Similarly let b be the lowest point among points to the 
left of irg (b may be equal to irg)- Rule (ii) applied to b and irg leads to the fourth diagram 
of Figure [15j Note also that we numbered two specific zones in this diagram and we study 
now the different cases where they are empty or not. 
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Figure 16: Zone 1 is non-empty 



Zone 1 is non-empty If zone 1 is non-empty, let x be the lowest point inside this zone (see 
Figure [TB]) . As there do not exist an increasing sequence GR, every point to the top-right of 
x is in G as shown in the second diagram of Figure [T6l where we define a zone 3. If zone 3 is 
empty then zone * is empty as a is G-indecomposable, hence every point has a assigned color 
as in the first diagram of Proposition 13.211 If zone 3 is non empty, let y be the rightmost 
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point inside this zone as shown in the third diagram. Applying rule (i) to x and y add another 
empty zone, leading to the last diagram. As a is Q-indecomposable, zone * is empty and all 
points have an assigned color as in the second diagram of Proposition 13.211 

Zone 1 is empty Suppose that zone 1 is empty. If zone 2 is also empty then as a is Q- 
indecomposable, zone * is also empty and all points have a determined color, as in the third 
diagram of Proposition 13.211 

If zone 2 is non-empty, let x be the rightmost point of zone 2. 
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Figure 17: Zone 1 is empty 



As there are no increasing subsequence GR, all points in the lower left quadrant of x lie 
in R as shown in the second diagram of Figure [17] where we define a zone 3. If zone 3 is 
empty then as a is Q-indecomposable zone * is also empty and all points have a determined 
color, as in the fourth diagram of Proposition 13.211 Otherwise zone 3 is non-empty and let 
y the lowest point in zone 3 as depicted in the third diagram. We apply rule (ii) to x and 
y leading to the fourth diagram. As a is 0-indecomposable, zone * is empty and all points 
have a determined color, as in the last diagram of Proposition 13.211 □ 

Definition 3.22. Let a be a permutation and i and j two indices of a such that OiOj is 
an ascent. Set a = max{/c | > o~j} and b such that = min{o"fc | k < i}. We define 
CRG(o~,i,j) as the partial bicoloring of a having the following shape: 
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where points of zone 3 are in G if zone 1 is empty and zone 2 is nonempty, 
in R if zone 1 is nonempty and zone 2 is empty, and have no color otherwise. 

Proposition 3.23. Let a be a Q-indecomposable permutation and c a valid coloring of a such 
that there is no increasing subsequence RG in c and there is at least an increasing sequence 
GR in c. Then c = Crg {°~ > ^RG > JRG ) ■ 



Proof: This is a direct consequence of Proposition 13.211 and Definition 13.221 



□ 



3.3.4 There exist both increasing sequences labeled GR and RG 

In this section we study the last case that remains to deal, i.e. there is at least one increasing 
sequence colored RG and at least one colored GR. As Aqr and Arg are non empty, icR-, 
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JGR, iRG and j RG are defined. We prove that once i GR , j G R, irg and j RG are determined, 
then it fixes the color of every other point of the permutation. 



Proposition 3.24. Let a be a permutation and c a valid coloring of a such that there exists 
at least an increasing sequence colored GR and at least an increasing sequence colored RG. 
Then c has one of the following shapes: 
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Proof: By maximality of i G R and minimality of (Xj GR we have: 
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By Theorem 13.131 we obtain: 
Recall that there exist an increasing sequence GR. i GR lies in quadrant 2 or 3 and j GR 
in quadrant 1 or 3. Hence the coloring c as either one of the 4 following shapes: 















Applying Theorem 13.141 to i GR and j GR we obtain these new diagrams: 
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Finally, using maximality of o~i GR and minimality of j GR , we obtain: 
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In the first 3 diagrams, the color of each point is determined - recall that upper-left and 
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lower-right points are determined by Theorems 13.131 and 13.141 - and only depend on irg, Jrg-> 
i G R and j GR . 

This leaves us with the last diagram of Figure [TH] for which we have again to consider 
several cases. Note that in this diagram we named several zones whose emptiness is relevant 
and we denote once more the unknown zone by *. 
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Figure 18: There exist increasing sequences labeled RG and GR 



Applying rule (vii) to Irg and jgr implies that zone C is empty. Similary rule (viii) 
applied to jgr and iqr proves that zone D is empty. If there exists a point x in zone A, then 
applying rule (ii) to jgr and x, all points in * are determined - they lie in R— as shown in 
the third diagram. Symetrically, if there exists a point y in B then applying rule (i) to y and 
iGR, ah points in * should be in G- see diagram 4 -. 

Thus this leaves us with the case where both A and B are empty. We show that this case 
is not possible. 
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Figure 19: A and S are empty 



A and i? are empty Then the permutation is colored as shown in the first diagram of 
Figure QjJl Let a be the lowest point among points to the left of irg (a may be equal to irg)- 
Rule (ii) applied to a and Irg implies the coloring shown in the second diagram - notice that 
if a = irg, the line between a and %gr does not exist -. Similarly, define b as the rightmost 
point among points above jgr (b may be equal to j'gr)- Rule (i) applied to b and jrg leads 
to the third diagram. At last we consider the topmost point c among points to the right of 
jgr (maybe c = jgr) and we apply rule (ii) to c and jgr- We also introduce d as the leftmost 
point among points below zqr (maybe d = i G r)- Rule (i) applied to d and igr leads to the 
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last diagram where different zones are numbered. We now study different cases according 
whether zone 1 is empty or not, and we prove that both are excluded. 

Zone 1 is empty Suppose that zone 1 is empty. As a is Q-indecomposable then zone 2 
must contain at least one point. Denote by x the rightmost point of this zone. Figure [20 
illustrates the proof. 
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Figure 20: Increasing sequences RG and GR exist and 1 is empty. 

Rule (vii) applied to x and c leads to the second diagram. Moreover as (igr, 3gr) is the 
topmost and leftmost increasing sequence GR, all points to the lower left quadrant of x lie in 
R, leading to the third diagram where we define a zone A. 

Zone 4 is not empty We prove that this case is not possible. If zone 4 is not empty, let y 
be its leftmost point (above or below jgr) as illustrated in the first diagram of Figure 
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Figure 21: Zone 1 is empty and zone 4 is not empty 

We apply rule (ii) to y and c and obtain the second diagram. But (zrg, Jrg) is lowest- 
right increasing sequence RG, hence there is no point labeled G in the above-right quadrant 
of y. Hence zone 3 is empty which is forbidden as a is 0-indecomposable. 

Zone 4 is empty We prove that this case is also not possible. Suppose that zone 4 is empty 
as illustrated in the first diagram of Figure [22j 
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Figure 22: Zone 1 is empty and zone 4 is empty. 



As a is 0-indecomposable, zone 3 is non-empty. Let z be the topmost point of zone 3 
(it may be to the left or to the right of igr)- Applying rule (i) to z and d we obtain the 
second diagram. But {Irg, 3rg) is t ne lowest right increasing sequence labeled RG, hence 
there are no point labeled R in the below-left quadrant of z - see diagram 3 -. But then a is 
0-decomposable which is forbidden. 

Zone 1 is not empty Suppose that zone 1 of Figure [19] is non empty. Define x as the 
lowest point of this zone as shown in the first diagram of Figure [23) 
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Figure 23: Zone 1 is not empty. 



Rule (viii) applied to x and d implies the second diagram. Moreover, as (igr, 3gr) is the 
leftmost-top increasing sequence labeled GR, all points to the top right of x are in G, leading 
to the last diagram. 

Zone 3 is not empty If zone 3 is not empty, let y be its topmost point (y may be to the 
left or to the right of igr) as pictured in Figure [241 
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Figure 24: Zone 1 is not empty and zone 3 is not empty. 

Rule (i) applied to d and y gives the second diagram. But (z_rg> 3rg) is the bottom- 
rightmost increasing sequence RG, hence no point in the lower left quadrant of y lies in R. 
Hence zone 4 is empty and a is 0-decomposable which is forbidden. 
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Figure 25: Zone 1 is not empty and zone 3 is empty 



Zone 3 is empty Figure [25] illustrates the proof. As a is G-indecomposable, zone 4 is not 
empty. Let y be the leftmost point inside zone 4 - either above or under jgr - as depicted in 
the second diagram. Rule (ii) applied to y and c leads to the third diagram. But {irc, Jrg) 
is the bottom-rightmost increasing sequence RG, hence no point of G lies in the top-right 
quadrant of y leading to the fourth diagram. So a is 0-decomposable which is forbidden. 
This ends the cases study, proving that zone A and B cannot be both empty. 

□ 

Definition 3.25. Let a be a permutation and i,j,k,£ four indices of a such that OiOj and 
o^ai are ascents. We define the partial bicoloring C*(o-,i,j,k,£) of a as follows. 
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If Oi, (j£, cifc and aj have a relative position corresponding to one of the above diagrams, then 
we define C*(cr, k, £) as the partial bicoloring of a having the corresponding shape, where 
in the first diagram points of zone 1 are in R if zone A is nonempty, in G if zones B is 
nonempty, and have no color otherwise. 

Otherwise C*(cr, i,j, k, £) is the partial coloring with no point colored. 

Proposition 3.26. Let a be a -indecomposable permutation and c a valid coloring of a 
such that there exist increasing sequences RG and increasing sequences GR in c. Then c = 
C*(a, IRQ , juG j ^GRi Jgr) ■ 

Proof: This is a consequence of Proposition 13.241 and Definition 13.251 noticing that if there 
exists a point x in zone A, then applying rule (ii) to £ and x, all points in zone 1 belong to R, 
and if there exists a point y in zone B, then applying rule (i) to y and k, all points in zone 1 
belong to G. □ 



3.4 A first polynomial algorithm 

Data: a a Q-indecomposable permutation (whose size is denoted n). 
Result: The set E of valid colorings of a 
for c bicoloring of a being one of 

c is unicolor R 

c is unicolor G 

c = C GR (a,i,j) or C RG (a,iJ) for i € [l..n] and j G [i..n] s.t. aj > Oi 
c = C*(a,i,j,k,£) fori £ [l..n] and j G [i..n] s.t. aj > and 
for k G [l..n] and £ G [k..n] s.t. > 

do 

| If all points of a are colored and c is valid then add c to E; 
end 

Algorithm 3: ColoringIndecomposablel(o") 

Proposition 3.27. Algorithm [5] compute in time 0(n 5 ) the set of valid colorings of any 
Q-indecomposable permutation a. 

Proof: Let a be a 0-indecomposable permutation of size n and c a valid coloring of a. Then 
from Propositions 13.161 13.201 13.231 and 13.261 c is either monochromatic, or CGR(cr,i, j) or 
CR G (cr,i, j) for some i G [l..n] and some j G [i..n] such that o~j > ai, or c = C*(a,i,j,k,£) for 
some i G [l-.n], some j G [i..n] such that aj > ai, some k G [l..n] and some £ G [k..n] such 
that at > <7fc. Thus c is computed by Algorithm [3] and added to E as it is valid. Conversely, 
each coloring added to E is a valid bicoloring of a. 

Now consider the complexity of Algorithm [3l There are C(n 4 ) colorings computed. Indeed 
there are two monochromatic colorings, 0(n 2 ) colorings Cgr{o~, i,j) or Crg{c, i,j) and C(n 4 ) 
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colorings C*(cr, i, j, k, I). Moreover the coloring is computed in linear time and checking if the 
coloring is valid is done in linear time using Proposition 13.101 Hence Algorithm [3] runs in time 
0(n 5 ). □ 



4 An optimal algorithm 

4.1 Rooting colorings 

In this section we show how each diagram of Propositions 13.171 13.211 and 13.241 can be rooted 
in a given point such that each point icRi^RG-, JGR and jua can be deduced from this one. 
Moreover, given a diagram we show how we can assign colors to points of the permutations 
lying in a colored zone of the diagram in linear time. 

Definition 4.1. Let a be a permutation and s € [l..|<r|]. We set 

Ci(<7, s) = Cgr(ct, s, t) where t = minjfe | k > s and a^ > a s } 
C^ipi s) = Cgr(ct, t, s) where t is such that at = max{(jfe | k < s and a^ < cr s } 
C%(g, s) = Crg(o-, s, t) where t is such that at = min{cjfe | k > s and a^ > a s } 
C±(a, s) = Crg(ct, t, s) where t = max{fc | k < s and a^ < a s } 

C 5 (a,s) = C*(a,p,q,t,s) with 

t = max{/c | k < u and a^ < a s } 
with u = max{k | k < s and aj~ > a s }, 
p = m&x{k | k < t and at < a^ < a s } and 
q such that a q = minjcrfc | t < k < u} 

Ce(a,s) = C*(a,p,q,s,t) with 

t = min{k | k > s and a^ > cr s }, 
u = max{A; [ k < t and a\. > at}, 
p = max{k | k < u and a s < a^ < &t} and 
q such that a q = minjcifc \ a^ > at and p < k < u} 

Cj(a,s) = C*(a, q,p,t, s) with 

t such that at = max{<jfc | k < s and a^ < a s }, 
u such that a u = min{<7fc | k < t and a^ > at}, 
p such that a p = min{o"fc | at > o u and t < k < s} and 
q = max{/c | k < p and a u < a^ < a p } 

Cs(a,s) = C*(a,p, q, s,t) with 

t = min{k | k > s and a^ > a s }, 
u such that a u = max{<7fc | a^ > at and k > t}, 
v = max{fc | k < u and a^ > cr u }, 
p = max{A; | k < v and at < a^ < a u } and 
q such that a q = min{cjfc \ ak > a u and p < k < v} 
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Cg(a, s) = C*(a,q,p,t, s) with 

t such that at = max{<7/% | k < s and a^ < a s }, 
u = min{£; | k < t and a^ < at}, 
v such that a v = min{o"fc | k < u and a^ > a u }, 
p such that a p = min{ak \ a^ > a v and u < k < t} and 
q = maxjfe | k < p and a v < a^ < a p } 

Proposition 4.2. Let a be a Q -indecomposable permutation and c a valid coloring of a which 
is not monochromatic. Then there exists s S [1.. |cr|] and m £ [1--9] such that c = C m (a,s). 

Proof: As c is not monochromatic, then from Proposition 13.161 a has at least a pattern 12 
which is not monochromatic. 

If there is no increasing subsequence RG in c then there is at least an increasing sequence 
GR in c. Thus from Propostion 13.201 c = Cqr {o~, %gr > JGr) ■ Moreover, c has one of the u 
shapes described in Proposition 13. 171 If the shape of c is one of the two first shapes, then jqr 
is the leftmost point in the upper-right quadrant of icR and c = Ci(a,icR)- Otherwise the 
shape of c is the third one and icR is the topmost point in the bottom-left quadrant of jcR 
thus c = C 2 (a,j GR ). 

If there is an increasing subsequence RG in c but no increasing sequence GR, then from 
Proposition 13.231 c = Crg^^rG-, Jrg)- Moreover c has one of the 5 shapes described in 
Proposition 13.211 If the shape of c is one of the three first shapes, then 2GR is the lowest 
point in the upper-right quadrant of iqr and c = Cs{a,iGR)- Otherwise the shape of c is one 
of the two last shapes and iqr is the rightmost point in the bottom-left quadrant of jgr thus 
c = C 4 (cr, j GR ). 

If there is an increasing subsequence RG and an increasing sequence GR in c, then from 
Proposition 13.261 c = C* (er, irq , jjiQ , igr, Jgr) ■ Moreover c has one of the 5 shapes described 
in Proposition 13.241 

If the shape of c is the first one, let u be the rightmost point in the top left quadrant of 
jGR (maybe u = Jrg)- Then applying rule (ii) to iGR and u, c has the following shape: 



Thus icR is the rightmost point on the left of u below jGR- Moreover 
iRG is the rightmost point on the topleft quadrant of iGR below jGR- 
Finally j'/jg is the lowest point on the right of icR and on the left of 
u. Hence c = C 5 (a,j G R,)- 



If the shape of c is the second one of Proposition 13.241 let u be the rightmost point in the 
top right quadrant of jRG (maybe u = jRc)- Prom rule (viii) applied to jnc and icR-, u < icR- 
Then applying rule (ii) to irg and jGR and applying rule (i) to jRG and u if u ^ j'rg, c has 
the following shape: 

Thus jgr is the leftmost point in the upper right quadrant of icR and 
u is the rightmost point in the upper left quadrant of jcR- Moreover 
ifiG is the rightmost point to the left of u, below jcR and above iGR- 
Finally, jnc is the lowest point in the upper left quadrant of jcR and 
to the right of iRG- Hence c = CQ(a,iGR)- 

If the shape of c is the third one of Proposition 13.241 let u be the lowest point in the lower 
left quadrant of iRG (maybe u = iRc)- From rule (vii) applied to iRG and jcR-, ®u > Qj GR - 
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Then applying rule (i) to icR and jrg and applying rule (ii) to u and irg if u 7^ iRG, c has 



the following 


; shape: 
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Thus ic-R is the topmost point in the lower left quadrant of jgr and u is 
the lowest point in the upper left quadrant of icR- Moreover jrg is the 
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lowest point above u, to the right of igr and to the left of jgr- Finally, 









i 3G1 


Irg is the rightmost point to the lower left of jrg and above u. Hence 









c = C 7 {a,j G R) 



If the shape of c is the fourth one of Proposition 13.241 let u be the topmost point to 
the upright quadrant of Jgr and v be the rightmost point to the top-right quadrant of jrg 
(maybe v = jrg)- Note that u is above irg as u is above x (u is the topmost point) which 
is above irg- Then applying rule (ii) to irg and u and applying rule (iii) to jrg and v if 
v ^ 3RG, c has the following shape: 
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Thus jgr is the leftmost point in the up right quadrant of igr- Point 
u is the topmost point in the upper right quadrant of jgr- Point v is 
the rightmost point in the upper left quadrant of u. Then irg is the 
rightmost point to the left of v, below u and above %gr- At last, jrg 
is the lowest point above u, to the right of irg and to the left of v. 
Hence c = Cs(cr,i G R) 



If the shape of c is the last one of Proposition 13.241 let u be the leftmost point in the 
lower left quadrant of zqr and v be the lowest point in the lower left quadrant of irg (maybe 
v = iRa)- Note that u is to the left of jrg as it is to the left of y (u is the leftmost point) 
and y is to the left of jrg- Then applying rule (i) to u and jrg and applying rule (ii) to v 
and irg if v ^ irg, c has the following shape: 
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Thus %gr is the topmost point in the lower left quadrant of jgr and u 
is the leftmost point in the lower left quadrant of igr- Moreover v is 
the lowest point in the upper left quadrant of u and jrg is the lowest 
point above v and to the right of u and to the left of igr- Finally, irg 
is the rightmost point in the lower left quadrant of jrq and above v. 
Hence c = C 9 (a,j G R) 

□ 



Proposition 4.3. Let a be a permutation, s S [1..|<t|] and m E [1..9]. Then we can compute 
C m (o~, s), test whether all points of a are colored and check whether C m (a, s) is valid in linear 
time w.r.t. \a\. 



Proof: Theorems EE] and EH and ED] 



□ 
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4.2 Algorithm and linear number of sortings for 0-indecomposable permu- 
tations 



Data: a a Q-indecomposable permutation 


Result: The set E of valid colorings of a 


for c bicoloring of a unicolor R or unicolor G do 




If c is valid then add c to E; 


end 




for s from 1 to \a\ do 




for m from 1 to 9 do 






c = C m (a, s); 






If all points of a are colored and c is valid then add c to E; 




end 


end 





Algorithm 4: ColoringlndecOptimal(cr) 



Given any point s in the permutation the Algorithm decides if the permutation can be 
colored in each possible case depicted in Propositions 13. 1713.211 and 13.241 Note that diagrams 
of Propositions 13.1713.211 and 13.241 depend on v points igr > iRG > 3GR j 3RG ■ Indeed, we prove 
in section 14.11 that any diagram can be rooted in one point - say irg f° r example - and from 
this points, we can find in linear time any other points - zgr, Jgr-, 3rg for instance -. Then, 
we color the permutations with respect to the different zones defined in the diagram. In this 
process, some points may be uncolored, meaning that they lie in empty zone of the diagram 
hence have to be rejected. At last, we have a coloring according to diagram and we have to 
check that this coloring is valid. 

Theorem 4.4. A Q-indecomposable permutation of size n has at most 9n + 2 valid colorings. 
Those colorings can be computed using Algorithm^ in time 0(n 2 ) which is optimal. 

Proof: This is a direct consequence of Propositions 14.21 and 14.31 except for the optimality. 
Proposition 14.51 below implies that the size of the set of valid colorings of the identity of size 
n is 2n 2 , proving the optimality. □ 

Proposition 4.5. For all n the identity of size n has exactly 2n valid colorings. 

Proof: Let a be the identity of size n. For all k between 1 and n let C RG (resp. C GR ) be the 
coloring of a such that for all i, a is in R (resp. G) if i < k and in G (resp. R) otherwise. 
Then it is straightforward to check using Proposition 13.111 that C RG (resp. C GR ) is a valid 
coloring of a. Conversely if c is a valid coloring of the identity, rules (Hi) and (iv) of IZg 
imply that there are at most one pair of consecutive points whoses colors are different. So c 
is some C RG or some C GR . □ 

The property of having a linear number of sortings is not a special case of the identity. 
Indeed there are some simple permutations that also have a linear number of sortings, as 
shown in the next proposition. 

Proposition 4.6. Permutations cr (n) = (2n - l)(2n - 3)(2n)(2n - 5)(2n - 2)(2n - 7)(2n - 
4). ..5836142 of size In have at least 2n — 3 valid colorings. 

Proof. To prove the result, we exhibit 2n — 3 colorings. We look at set of four points of a 
whose indices (resp. values) are consecutive and which form a pattern 2413 (resp. 3142). 
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Notice that they can be taken to be {irg^gr, Jrg, Jgr} m a valid coloring of a respecting 
to the last (resp. third) diagram of Proposition 13.241 as shown in the figure below. This way 
we obtain 2n — 3 valid colorings of a. 




□ 



4.3 Final algorithm 

Recall first that if a permutation is 0-decomposable, then it is 2-stack pushall sortable if and 
only if each of the block of its decomposition is 2-stack pushall sortable and that we can just 
push elements of the first block according to any sorting procedure of it, then elements of the 
second and so on, before popping out all the elements. This means that the different colorings 
for a Q-decomposable permutation is the product of all colorings for each block. 

Proposition 4.7. Let a be a 'permutation and Col(a) the set of valid colorings of a. If 
a = 0[7Ti, . . . , 7Tfc] then the map c — > (c\iri, . . . , c\itk) is a bisection from Col{a) into Col(-Ki) x 

• • • X Col(lTk). 

Proof: Let c be a valid coloring of a, then c avoids patterns 132, 213, 2/13 and 1X2. Thus for 
all i, c\iii avoids patterns 132, 213, 2/13 and 1X2 hence is a valid coloring of 7Tj. Conversely 
let Ci £ Col(iri) for all i. Then coloring points of a according to (ci, . . . c&) {i.e. according to 
c\ for the \tt\\ first points of a, according to C2 for the following points and so on) leads 
to a coloring c of a which is valid. Indeed assume that c is not valid. Then c has a pattern 
132, 213, 2/13 or 1X2. Let p be such a pattern. Then p is not inside a block 7Tj as Cj is a valid 
coloring for all i. If all points of p are in different blocks 7Tj then p is 321 which is excluded. 
Thus there are one point of p in a block 7Tj and two points of p in a block ttj. If % < j then 
p begins with its greatest point, which is excluded as p is 132, 213, 2/13 or 1X2. If i > j 
then p ends with its smallest point, which is excluded as p is 132, 213, 2/13 or 1X2. As a 
consequence such a pattern p does not exists and c £ Col (a), concluding the proof. □ 

Data: a a permutation 

Result: A linear description of the set Col(a) of valid colorings of a 
Compute the 0-decomposition of a: a = Q[tti, . . . , irk] with 7Tj Q-indecomposable; 
for i from 1 to k do 
| Compute Col(iTi) thanks to Algorithm HJ 
end 

Return (CoI(tti), . . . , Col^k))] 

Algorithm 5: Colorings (a) 

Proposition 4.8. Let a be a permutation of size n. Then Algorithm^ gives a linear descrip- 
tion of Col(o-) in time 0(n 2 ). 

Proof. The algorithm computes the ©-decomposition of a: a = Q [iri , . . . , 7Tfc] with 7Tj 0- 
indecomposable. This is done in linear time. If k = 1 then a is 0-indecomposable and 
Col(a) = Col{-K\). We concludes thanks to Theorem 14.41 If k > 1 then from Proposition 14.71 
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Col{o~) ~ CoI{tt\) x • • • x CoI(tt^). For all i, CoKjTj) has a size is smaller than 9|7q| and is 
computed in 0(|7r| 2 ). We concludes the proof noticing that 9|7i"i| + • • • + 9|7Tfc| = 9|<r| and 

IvTxl 2 + • • • + |7r fc | 2 < |(j| 2 . □ 

Theorem 4.9. Using Algorithm^ we can decide in time 0(n 2 ) whether a permutation a of 
size n is 2-stack pushall sortable. 

Proof. By Theorem 13,91 a permutation a is 2-stack pushall sortable if and only if it admits a 
valid coloring. Thus all we need is to test whether each set Col^i) returned by Algorithm [5] 
is non-empty with a = Q[iti, . . . , irk] being the Q-decomposition of a, and we conclude using 
Proposition [OJ □ 



5 Conclusion 



This article defines a new restriction of 2-stacks sorting, namely 2-stacks pushall sorting. We 
characterize every possible pushall sorting of a permutation by means of a bi-coloring of the 
permutation. Then we give an 0(n 2 ) algorithm which computes a linear representation of 
all pushall sortings of a given permutation, which thus decides if a permutation is 2-stack 
pushall sortable. We proove that this complexity is optimal. 

More studies remain to be done on 2-stacks pushall sorting. First, a simpler mathematical 
characterization of 2-stack pushall sortable permutations would be interesting. Then, we could 
study more in depth the number of pushall sortings of a given permutation. More generally 
it would be nice to compute the generating function of 2-stack pushall sortable permutations, 
or at least asymptotic bounds on this function. But most importantly, this result is a step to 
the solve the general 2-stack sorting, which we do in a forthcoming article. 
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